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(57) The electronic secretary system comprises an 
input section (101) for inputting operation information; a 
processing section (102) for performing a predetermined 
processing in accordance with the input infomiation; a 
data storing section (103) for storing the input informa- 
tion; an internal state memory section (104) for memo- 
rizing the state of processing; an action description stor- 
ing section (106) for storing the action description of an 
electronic secretary:.an action data storing section (107) 
for storing data, such as an action image, a voice, or the 
like of the electronic secretary: an electronic secretary 
generation section (105) for deciding and generating an 
action of the electronic secretary in accordance with the 
internal state; a display section (108) for displaying an 
output of the processing section (102) and an output of 
the electronic secretary generation section (105); an in- 
terface (109) for transmitting data with other equipment; 
and an interpreter section 1 10 for interpreting the trans- 
mitted data. 
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Description 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention refers to an electronic secre- 
tary system and more particularly, to an electronic sec- 
retary system for performing a series of processing 
through an interaction between a user and an electronic 
secretary in an application operating on a plurality of 
equipment connected through a communication net- 
work. 

2. Description of the Related Art 

Formerly, there are some applications, such as com- 
putation and word processor, with which a person image 
or other images is displayed and a voice is generated to 
improve a man-machine interface. 

As an improvement in man-machine interface by us- 
ing an autonomously acting agent, there is a virtual living 
being system disclosed in Japanese Patent Laid-Open 
Publication No. 68063/1994. This stores a knowledge 
needed for a virtual living being in autonomously acting 
into a knowledge database, while a virtual living being 
simulates an action based on the knowledge stored in 
the knowledge database and the internal state of a virtual 
living being stored in the internal state memory means, 
and sends out the results to a drawing system, voice sys- 
tem, or the like. 

Systems for pertorming an application by using a 
plurality of equipment connected through a network in- 
clude, e.g., a schedule management system described 
in ..Japanese Patent Laid-Open Publication No, 
6378/1 993, but this invention refers to the internal mech- 
anism and describes nothing about a human interface. 

When making an information processing equipment 
perform a series of processing, especially when manip- 
ulating an application operating in a plurality of equip- 
ment through a communication network, a conventional 
system has been hardly manageable because what 
processing is performed inside becomes difficult for a 
user to understand. 

To solve these problems, the present invention pro- 
vides an electronic secretary system that can achieve an 
improvement in man-machine interface by display the 
contents of processing or internal state to a user in an 
easily understandable form through an action of an elec- 
tronic secretary imitating a person to operate various 
processing. 

SUfVIMARY OF THE INVENTION 

According to an aspect of the present invention, 
there is provided an electronic secretary system for per- 
forming a series of processing through an interaction be- 
tween a user and an electronic secretary displayed as 



an object imitating a human action, comprising: input 
means for inputting information, such as user's instruc- 
tion: processing means responsive to the input informa- 
tion for performing a predetermined processing: data 

5 storing means for storing data to be processed: internal 
state memory means for memorizing the internal state 
of processing: first action description storing means for 
storing the action of said electronic secretary described 
corresponding to the state of said processing; first elec- 

'0 tronic secretary generation means for generating actions 
of said electronic secretary from the state of processing 
memorized in said internal state memory means and the 
description stored in response to the state of processing 
into said action description storing means; and display 

'5 means for performing an action of said electronic secre- 
tary and a display of data. 

According to an aspect of the present invention, 
there is provided an electronic secretary system further 
comprising: means for communicating with other one or 

20 more electronic secretary systems: second action de- 
scription storing means for storing data describing the 
action of another electronic secretary in a communica- 
tion partner; and second electronic secretary generation 
means for generating an action of said another electronic 

25 secretary in the communication partner. 

According to an aspect of the present invention, 
there is provided an electronic secretary system further 
comprising: mode memory means for memorizing the 
user's work environment. 

30 According to an aspect of the present invention, 
there is provided an electronic secretary system further 
comprising: internal state generation means for gener- 
ating the internal state in accordance with a predeter- 
mined rule. 

35 According to an aspect of the present invention, 
there is provided an electronic secretary system com- 
prising: retrieval means for retrieving data corresponding 
to the retrieval condition inputted by a user from the elec- 
tronic secretary system of said user himself or other elec- 

4p tronic secretary systems connected through a commu- 
nication means instead of said processing means. 

According to an aspect of the present invention, 
there is provided an electronic secretary system wherein 
said processing means performs the registration, dele- 

^5 tion or retrieval of a schedule in the electronic secretary 
system of said user himself or other electronic secretary 
systems connected through a communication means. 

According to an aspect of the present invention, 
there is provided an electronic secretary system further 

50 comprising: history memory means for memorizing the 
history of the interna! state: wherein in response to re- 
quest from a user, what action the electronic secretary 
has made is displayed on said display means by referring 
to the history memorized on said history memory means. 

55 According to an aspect of the present invention, 
there is provided an electronic secretary system further 
comprising: communicator means for communicating 
with other one or more electronic secretary systems; 
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electronic mail processing means tor processing elec- 
konic mail letters transferred through said communicator 
means: and classifying means tor classifying the elec- 
tronic mail processed by. said electric mail means. 

According to an aspect of the present invention. 5 
there is provided an electronic secretary system further 
comprising: classifying mode memory means for mem- 
orizing a classifying mode by referring to the instruction 
inputted from said input means. 

According to the electronic secretary system of the io 
present invention, since it is so arranged that first action 
description storing means stores the action description 
of an electronic secretary displayed as an object imitat- 
ing a human action, an instruction or the tike is inputted 
by input means, a predetermined processing is per- fS 
formed in response to input information by processing 
means, the state of processing is memorized by internal 
state memory means, and an action of the electronic sec- 
retary corresponding to the internal state is determined 
out of the action description, and generated by first gen- 20 
eration means, and outputted to the display means, a 
user can get the electronic secretary system execute a 
series of processing through an interaction with the elec- 
tronic secretary, thereby enabling an improvement in 
man -machine interface. 25 

According to the electronic secretary system of the 
present invention, since it is so arranged that second ac- 
tion description storing means stores data describing the 
actions of electronic secretaries in one or more commu- 
nication partners and second generation means gener- 30 
ates an action of each partner electronic secretary, an 
action of the partner electronic secretary is displayed, 
thereby enabling a user to identify the internal state of 
communication partners. 

According to the electronic secretary system. of the 3S 
present invention, since it is so arranged that mode 
memory means memorizes the user's work environment 
and data stored in electronic secretary action data stor- 
ing means are switched by electronic secretary genera- 
tion means in response to a setting value stored in said 
mode memory means, an output is obtained which cor- 
responds to the work environment or the user's setting. 

According to the electronic secretary system of the 
present invention, since it is so arranged that the internal 
state is generated by internal state generation means in -^s 
accordance with a predetermined rule and written into 
the internal state memory means and that an action of 
the electronic secretary corresponding to the internal 
state is generated by electronic secretary generation 
means, actions, such as wink, of the electronic secretary so 
can be actualized and an autonomous action becomes 
possible. 

According to the electronic secretary system of the 
present invention, since it is so arranged that data cor- 
responding to the retrieval condition inputted by a user ss 
are retrieved by retrieval means from the data storing 
means distributed in a plurality of equipment connected 
through a communication means, notice can be given to 
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a user that the system is engaged in retrieval by an elec- 
tronic secretary exhibiting the process of performed re- 
trieval to a user. 

According to the electronic secretary system of the 
present invention, since it is so arranged that the regis- 
tration, deletion or retrieval of schedule is performed by 
said processing means and the registration, reference 
or the like of schedule data is performed through com- 
munication means notice can be given using an electron- 
ic secretary to a user of the actual state of the registra- 
tion, deletion or retrieval of schedule. 

According to the electronic secretary system of the 
present invention, since it is so arranged that with the 
history of the internal state memorized in the history 
memory means, what action has been made in the past 
by an electronic secretary is displayed on the display 
means by request of a user, a user can regenerate a mo- 
tion of the electronic secretary and thus can identifies 
what action has been made by the electronic secretary. 

According to the electronic secretary system of the 
present invention, since it is so arranged that data are 
trarisferred by communicator means with other equip- 
ment connected through a network and that the electron- 
ic mail letter transferred is processed by electronic mail 
processing means, data of the electronic mail processing 
means and an action of the electronic secretary are dis^ 
played on display means and electronic mail letters are 
classified by classifying means when data comprising 
electronic mail, a great many electronic mail letters can 
be classified by a user's desired setting even if arriving 
through a communication network, enabling the ease of 
using an application to be improved. 

According to the electronic secretary system of the 
present invention, since it Is so arranged that a classify 
mode based on the instruction inputted from said input 
means is memorized by classify mode memory means, 
a user can easily set the classify mode at any time. 

Further objects and advantages of the present in- 
vention will be apparent from the following description of 
the preferred embodiment of the invention as illustrated 
in the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 



Figure 1 is a block diagram showing a first embodi- 
ment of an electronic secretary system according to the 
present invention. 

Figure 2 is a data list showing the electronic-secre- 
tary's actions corresponding to various internal states. 

Figure 3 is a flowchart showing operations of a first 
embodiment of an electronic secretary system according 
to the present invention. 

Figure 4 is a flowchart showing operations of a first 
embodiment of an electronic secretary system according 
to the present invention. 

Figure 5 is a progress chart showing operations of 
a first embodiment of an electronic secretary system ac- 
cording to the present invention. 
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Figure 6 is a flowchart showing operations oi a first 
embodiment of an electronic secretary system according 
to the present invention. 

Figure 7 is a progress chart showing operations of 
a first embodiment of an electronic secretary system ac- 5 
cording to the present invention. 

Figure 8 is a flowchart showing operations of a first 
embodiment of an electronic secretary system according 
to the present invention. 

Figure 9 is a block diagram showing a second em- 
bodimenl of an electronic secretary system according to 
the present invention. 

Figure 10 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 11 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 12 is to illustrate a practical display image of 
an electronic secretary system according to the present 20 
invention. 

Figure 1 3 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 14 is to illustrate a practical display image of 25 
an electronic secretary system according to the present 
invention. 

Figure 15 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 16 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 17 is to illustrate a practical display image of 
an electronic secretary system according to the present 3S 
invention. 

Figure 18 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 1 9 is to illustrate a practical display image of -to 
an electronic secretary system according to the present 
invention. 

Figure 20 is a flowchart showing operations of a sec- 
ond embodiment of an electronic secretary system ac- 
cording to the present invention. . 

Figure 21 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 22 is a block diagram showing a third embod- 
iment of an electronic secretary system according to the 50 
present invention. 

Figure 23 is a block diagram showing a fourth em- 
bodiment of an electronic secretary system according to 
the present invention. 

Figure 24 is a block diagram showing a fifth embod- ^5 
iment of an electronic secretary system according to the 
present invention. 

Figure 25 is a block diagram showing a sixth embod- 
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iment of an electronic secretary system according to the 
present invention. 

Figure 26 is a flowchart showing operations of a sixth 
embodiment of an electronic secretary system according 
to the present invention. 

Figure 27 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 28 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 29 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 30 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 31 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 32 is a block diagram showing a seventh em- 
bodiment of an electronic secretary system according to 
the present invention. 

Figure 33 is to illustrate a practical display image of 
an electronic secretary system according to the present 
invention. 

Figure 34 is a block diagram showing an eighth em- 
bodiment of an electronic secretary system according to 
the present invention. 

Figure 35 is a data list showing the electronic -sec- 
retary's actions corresponding to various internal states. 

Figure 36a is a flowchart showing operations of an 
eighth embodiment of an electronic secretary system ac- 
cording to the present invention. 

Figure 36b is a flowchart showing operations of an 
eighth embodiment of an electronic secretary system ac- 
cording to the present invention. 

Figure 37 is a flowchart showing operations of an 
eighth embodiment of an electronic secretary system ac- 
cording to the present invention. 

Figure 38 is a flowchart showing operations of an 
eighth embodiment of an electronic secretary system ac- 
cording to the present invention. 

Figure 39 is to illustrate an electronic mail letter. 

Figure 40 is to illustrate the categories for classify- 
ing. 

Figure 41a is to illustrate a similarity for each cate- 
gory. 

Figure 41 b is to illustrate the extracted result of sim- 
ilarity. 

Figure 41c is to illustrate displaying examples of 
headers of electronic mail letters. 

Figure 41 d is to illustrate displaying examples of 
headers of electronic mail letters. 

Figure 42 is to illustrate practical display images of 
an electronic secretary system according to the present 
invention. 

Figure 43 is to illustrate practical display images of 
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an electronic secretary system according to the present 
invention. 

Figure 44 is to illustrate practical display innages of 
an electronic secretary system according to the present 
invention. 5 

Figure 45 is to illustrate practical display images of 
an electronic secretary system according to the present 
invention. 

Figure 46 is to illustrate practical display images of 
an electronic secretary system according to the present 'O 
invention. 

Figure 47 is to illustrate practical display images of 
an electronic secretary system according to the present 
invention. 

Figure 48 is a block diagram showing a ninth em- is 
bodiment of an electronic secretary system according to 
the present invention. 

Figure 49 is a flowchart showing operations of a 
ninth embodiment of an electronic secretary system ac- 
cording to the present invention. 20 

Figure 50 is to illustrate various classify modes. 

Figure 51 is to illustrate estimation functions corre- 
sponding to individual classify modes. 

Figure 52a is to illustrate displaying examples for 
each classify mode. 2S 

Figure 52b is to illustrate displaying examples for 
each classify mode. 

Figure 52c is to illustrate displaying examples for 
each classify mode. 

30 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

Hereinafter, a first embodiment of an electronic sec- 
retary system according to the present invention will be ^5 
described referring to the drawings. 

As shown in Figure 1 , an electronic secretary system 
according to the present invention comprises: an input 
section 101 as input means, such as mouse and key- 
board, for inputting operational information; a processing -to 
section 1 02 as processing means responsive to the input 
information for performing a predetermined processing; 
a data storing section 103 for storing input information; 
an internal state memory section 104 as internal state 
memory means for memorizing the internal state of . -^5 
processing; an action description storing section 106 as 
first action description storing means for storing the ac- 
tion description of an electronic secretary as an object 
imitating a human action: an electronic secretary action 
data storing section 107 as first data storing means for so 
storing data, such as image of an action and voice, of 
the electronic secretary; an electronic secretary genera-, 
tion section 1 05 as action decision means for deciding 
and first generation means for generating an action of 
an electronic secretary corresponding to the internal ss 
state out of the action description: a display section 108 
for displaying an output of the processing section 102 
and an output of the electronic secretary generation sec- 
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tion 105: a communication interface 109 for performing 
data transfer with other equipment connected through a 
network; and an interpreter section 110 for interpreting 
data to be transferred and performing data transfer be- 
tween the processing section 102 and communication 
interface 109. 

Data describing an action of the electronic secretary 
stored in the action description storing section 106 cor- 
responding to the internal state are as shown in Figure 
2. The internal stale shown in Figure 2 is actually stored 
together with related data added thereto. The related 
data include, for example, names of communication part- 
ners or contents of retrieval requests, and are auxiliary 
information for actions of the electronic secretary, for ex- 
ample, information on XX for performing an action of 
"There was a request from XX.* 

Next, actions of the electronic secretary generation 
section 105 will be described along the flowchart of Fig- 
ure 3. 

The internal state written in the internal state mem- 
ory section 104 is read by electronic secretary genera- 
tion section 105 (step SI), while the action description 
corresponding to the internal state is read out from data 
stored in the action description storing section 106 (step 
S2). Based on the action description read out, an action 
of the electronic secretary is generated (step S3), and a 
voice, image or the like stored in the electronic secretary 
action data storing section 1 07 is outputted to the display 
section 108 (step S4). These actions are repeated at 
each elapse of a predetermined time. 

Next, actions in the case of retrieval will be described 
along the flowchart of Figure 4. 

When a user selects a "Retrieval- button by using a 
mouse or the like (step S11). "Retrieval Start" is written 
in the internal state memory section 104 by the process- 
ing section 102 (stiep S12) and at the same time a win- 
dow for keyword input is opened. When "Retrieval Start" 
is written in the internal state rnemory section 104, an 
action of the electronic secretary corresponding, to "Re- 
trieval Start" is read out from data of Figure 2 stored in 
the action description storing section 106, and voice data 
"Input a keyword", stored in the electronic secretary ac- 
tion data storing section 107 is regenerated. At this time, 
the mouth of the electronic secretary moves correspond- 
ing to an output of voice data, where the movement of 
the mouth is implemented by an alternate display of im- 
ages of closed mouse and opened mouth during voice 
output. From this on, similarly when a voice is outputted. 
the mouth of the electronic secretary moves during voice 
output. Incidentally, the number of internal states may be 
plural. For example, it can also be set that a state of re- 
trieval start sets in while keeping a state of request re- 
ceipt. This means that the system stands in an allowable 
state of receiving a request from a user and the electronic 
secretary performs an action of retrieval start. 

When a user inputs a keyword by using a keyboard 
(step Si 3), the processing section 102 decides whether 
the "Break" button was pressed or not (step Si 4). If the 
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'Break" button is pressed, the retrieval processing is bro- 
ken off. If deciding that the "Break' button is not pressed, 
the processing section 102 decides whether the "Re- 
trieval" button is pressed (step SI 5). If the 'BetrievaT 
button is not pressed, the systenn returns to the step S 1 4. 
And if the 'Retrieval" button is pressed. "Partner Selec- 
tion" is written in the internal state storing section 104 
(step Si 6). When "Partner Selection" is wfiiten= "Whom 
do you ask?" is regenerated as an action of the electronic 
secretary from data of Figure 2. A "Name" button of the 
partner to be asked for retrieval is selected using a 
mouse or the like by a user (step 817). A mark is placed 
on the people selected and it is decided whether the 
"Break" button is pressed or not (step 818). If the "Break" 
button is pressed, the retrieval processing is broken off. 
Or. if the "Break" button is not pressed, it is decided 
whether the "OK" button is pressed (step Si 9). If the 
"OK" button is not pressed, the system returns to the step 
SI 8. Or. if the "OK" button is pressed, a communication 
with other equipment starts (step S20). 

Then, when the transmit processing of requests for 
ail selected communication partners ends (step S21). 
the processing of retrieval starts (step S22). Retrieval 
can be executed, for example, by a method for extracting 
a sentence in which any of keywords of sentence data 
stored in the data storing section 103 and a keyword in- 
putted by a user is coincident or similar. During retrieval. 
"During Retrieval" is written into the internal state storing 
section 104. When the internal state is "During Retriev- 
al", an action of leafing through a book is performed as 
an electronic secretary's action from data of Figure 2. 
This processing is performed for all data till the retrieval 
is completed (step 323). 

Here, the contents of communication with other 
equipment will be described in detail. 

By way of example, a comniunication between 
Equipment A (user's name: Ueda) and Equipment B (us- 
er's name: Sakamoto) will be described. The contents of 
communication is outputted from the processing section 
102 of Equipment A to the interpreter section 110. The 
information of a communication partner and that of the 
contents of communication are converted into commu- 
nication codes by the interpreter section 1 1 0 and output- 
ted to the communication interface 109. Actual transfer 
of data is performed by the communication interface 1 09. 
As shown in Figure 5, the internal state in Equipment A 
and Equipment B, an action of the electronic secretary, 
and transmit data are provided. First, a request for com- 
munication start from Equipment A to Equipment B is 
transmitted. At this time, the internal state becomes 
"Communication Start Transmit"^ the electronic secre- 
tary performs an action of telephoning in accordance 
with Figure 2. In Equipment B. data of communication 
start request are received from Equipment A, the internal 
state becomes "Communication Start Receive", and 
data of "OK" is transmitted as a reply. At this time, the 
electronic secretary of Equipment B performs an action 
of receiving a telephone and a voice of "Hello, this is 



(Sakamoto's) secretary speaking." is regenerated. Inci- 
dentally, the name in a bracket is a user's name set for 
each equipment and differs with equipment. 

In Equipment A that has received data of OK, the 
5 internal state becomes "Request Transmit" and the con- 
tents of request is transmitted. In the case of retrieval, 
the contents of request comprise keyword, name of the 
retrieval requesting user ("Ueda" in this example), name 
of the retrieval result output file, and the like. The format 
'0 is, for example, as follows: 

Search ("Computer Kagaku", "Ueda". "ken- 
saku.txf). where Search means a request of retrieval. 

At this time, the electronic secretary in Equipment A 
regenerates "This is (tJeda's) secretary speaking and 

15 there is a request from (Ueda)". The internal state of 
Equipment B that has received the contents of request 
becomes "Request Receive" and OK data is transmitted 
as a reply. At this time, the electronic secretary in Equip- 
ment B regenerates "I've understood and 1*11 tell 

20 (Sakamoto). The internal state of Equipment A that has 
received OK data becomes "Communication End Trans- 
mit" and a request of communication end is transmitted. 
The internal state of Equipment B that has received a 
request of communication end becomes "Communica- 

25 lion End Receive" and the electronic secretary of Equip- 
ment B performs an action of hanging up. 

After the above series of communication ends, the 
contents of request are executed. In this example, the 
retrieval is executed. 

30 Next, a series of actions: receiving a request through 
communication, then executing the request, till returning 
the result of the request will be described along the flow- 
chart of Figure 6. 

Keep waiting till a request from other equipment oc- 

35 curs (step S25). When a request occurs, receive the re- 
quest following the above procedure (step 26) and exe- 
cute the request (step S27). Then, transmit the result of 
the request to the equipment of the user that has ad- 
vanced the request (step S28). Furthermore, turn the in- 

•^0 ternal state into 'Request End". At this time, the electron- 
ic secretary performs an action of raising her hands from 
data of Figure 2. 

Next, the procedure of communication concerning 
the reply of request result between Equipment B and 

•^5 Equipment A will be described using Figure 7. 

First, the internal state of Equipment B becomes 
"Communication Start Transmit" and a request for com- 
munication start is transmitted to Equipment A. The in- 
ternal state of Equipment A that has received a request 

50 for communication start becomes "Communication Start 
Receive" and data of OK is transmitted as a reply. The 
internal state of Equipment B that has received OK data 
becomes "Retrieval Result Transmit". The electronic 
secretary of Equipment B regenerates "We'll transmit the 

55 retrieval result" and the content of request result is trans- 
mitted. The transmitted content contains the header part 
and text of retrieved sentences, similarity of each sen- 
tence and the keyword, name of transmitting user and 
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the like. The internal stale of Equipment A that has re- 
ceived the retrieval result becomes 'Retrieval Result Re- 
ceive" and data of OK is transmitted as a reply. At this 
time, the electronic secretary of Equipment A regener- 
ates "We're obliged." Next, the internal state of Equip- 
ment 8 that has received OK data becomes "Communi- 
cation End Transmit" and a request of communication 
end is transmitted. The request of communication end is 
received by Equipment A and the internal state becomes 
"Communication End Receive". 

Next, the processing procedure of equipment when 
transmitting the requirements of a request and waiting 
the request result will be described along the flowchart 
of Figure 8. 

Keep waiting till a request of communication occurs 
(step S30). When a request of communication occurs, 
receive the request result as mentioned above (step 
S31). And. repeat the above processing till the request 
results return from all equipment to which the request 
are transmitted (step S32). When the results are trans- 
mitted from all equipment, the internal state becomes 
"Retrieval End" (step S33). 

Next, a second embodiment of an electronic secre- 
tary system according to the present invention will be de- 
scribed referring to the drawings. 

As shown in Figure 9. an electronic secretary system 
according to the present invention comprises: an input 
section 101 . such as mouse and keyboard, for inputting 
operational information: a processing section 102 re- 
sponsive to the input information for performing a prede- 
termined processing; a data storing section 103 for stor- 
ing input Information; an internal state memory section 
104 for memorizing the internal state of processing; an 
action description storing section 106 for storing the ac- 
tion description of an electronic secretary as an object 
imitating a human action: an electronic secretary action 
data storing section 107 for storing data, such as image 
of an action and voice, of the electronic secretary; an 
electronic secretary generation section 105 for deciding 
and generating an action of the electronic secretary cor- 
responding to the internal state out of the action descrip- 
tion; a display section 108 for displaying an output of the 
processing section 102 and an output of the electronic 
secretary generation section 105; a communication in- 
terface 1 09 for performing data transfer with other equip- 
ment connected through a network; an interpreter sec- 
tion 110 for interpreting data to be transferred and per- 
forming data transfer between the processing section 
102 and communication interface 109; a second action 
description storing section 11 2 as second action descrip- 
tion storing means for storing data describing the action 
of the electronic secretary in one or.more communication 
partners; a second electronic secretary action data stor- 
ing section 1 1 3 as second data storing means for storing 
data, such as image of an action and voice, of the elec- 
tronic secretary; and a second electronic secretary gen- 
eration section 1 1 1 as second generation means for gen- 
erating an action of the electronic secretary. 



Ne)ct, actions during communication will be de- 
scribed. 

As shown in Figure 5, when advancing a request 
from Equipment A to Equipment B, the internal state of 

5 Equipment A becomes "Communication Start Transmit" 
and a request of communication start is transmitted to 
Equipment B. At this time, not only a request of commu- 
nication start but also the user's name and internal state 
of Equipment A on the transmitter side are transmitted. 

10 In the interpreter section 110 of Equipment B on the re- 
ceiver side, a request of communication start is outputted 
to the processing section 102 and the user's name and 
internal slate, information related to the electronic sec- 
retary of Equipment A is outputted to the second elec- 

'5 tronic secretary generation section 111. In the second 
electronic secretary generation section 111. the action 
description of an electronic secretary corresponding to 
the internal state and data on an action of the electronic 
secretary responsive to the user's name are read from 

20 the second action description storing section 112 and 
second electronic secretary action data storing section 
113 respectively, and actions of the electronic secretary 
are generated. 

Here, with respect to data on a second action de- 

25 scription and second electronic secretary action data, it 
is considered that at the start of corhmunication data are 
previously prepared for a plurality of communication 
partners, action describing data and second electronic 
secretary action data are selected responsive to a com- 

30 munication partner, and an action of another electronic 
secretary is generated, or that at the start of communi- 
cation, data on a second action description and second 
electronic secretary action data are transmitted and 
stored in a second action description storing section 1 1 2 

3S and second electronic secretary action data storing sec- 
tion 1 1 3 at the destination, respectively. 

Next, the above-mentioned example of actions will 
be described referring to an actual display screen. 

As shown in Figure 10, the initial screen of the dis- 

-fo play displays a schedule button 1 . mail button 2. retrieval 
button 3, data button 4 and note button 5 which corre- 
spond to individual processing at the lower left, and an 
electronic secretary 6 and secretary button 7 on the right. 
In the middle of the screen, there are a button 21 mate- 

-^5 rializing a calendar, a button 22 materializing a mailbox, 
a button 24 materializing a data file, and a button 25 ma- 
terializing a notebook, which perform the respective 
processing similar to that of the schedule button 1, mail 
button 2. data button 4 and note button 5. Here, when 

50 pressing the retrieval button 3. a window 8 for keyword 
input as shown in Figure 1 1 comes to appear and a user 
inputs a keyword from the keyboard and the like. Here, 
when pressing the retrieval button in the window 8. a win- 
dow 9 for displaying the names of those whose request 

55 is asked comes to appear as shown in Figure 1 2. Then, 
when pressing a name button, a mark for indicating a 
selection of the relevant person is put on the name button 
pressed. In the illustrated example, "Self", "Sakamoto" 
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and "Matsuoka" are selected. Then, when pressing the 
OK button 10. a communication with those selected 
above starts. Figure 13 shows an example of a commu- 
nication with the electronic secretary 1 1 o( "Sakamoto'*. 
An action of the electronic secretary of another equip- 
ment is displayed on the upper right screen. Figure 14 
shows an example of a communication with the electron- 
ic secretary 12 of "Matsuoka". 

After the communication with other equipment ends, 
a processing of retrieval starts. Figure 15 shows a 
book-leafing action of an electronic secretary engaged 
in retrieval. Figure 16 shows an example of receiving the 
retrieved result. Figure 1 7 shows an action of an elec- 
tronic secretary when the request results are returned 
from all equipment and the retrieval ends. Here, when 
pressing the secretary button 7, a window 13 for the list 
of retrieved results Is displayed as shown in Figure 18. 
The header part 14 of the retrieved sentences, similarity 
1 5 to retrieved keywords and name 1 6 of returning users 
of request results is respectively displayed in the window 
13. Then, when pressing a button 17 in the header part 
of sentences, a window 1 8 having the text of a sentence 
written thereon is displayed as shown in Figure 19. 

Next, an action performed when a user pressed the 
secretary button 7 will be described along the flowchart 
of Figure 20. 

When a user presses the secretary button 7 (step 
S40). the internal state is checked (step S41). If the in- 
tema! state is "Retrieval End*, the internal state is anew 
turned into "Retrieved Result" (step S42). An action de- 
scription is read. that corresponds to the internal state 
"Retrieved Result". "The retrieved results are ready" is 
regenerated and at the same time the retrieved results 
are displayed (step S43). Or. if the internal state is "Re- 
quest End", the internal state is anew turned into "Re- 
quest Reply" (step S44). At this time, an action descrip- 
tion is read that corresponds to the internal state "Re- 
quest Reply", "There has been a request (for retrieval) 
from Mr (Ueda)" is regenerated and the requested re- 
sults are displayed (step S45). On the other hand, if the 
internal state is otherwise, the internal state is anew 
turned into "Help" (step S46). An action description is 
read that corresponds to the internal state "Help" and a 
help screen as shown in Figure 21 is displayed (step 
S47). In this example, the state of the self equipment and 
communication partner equipment is indicated by anima- 
tion, voice and the like, but may be indicated only by sen- 
tences If they represent the state of self and partners. 

Next, a third embodiment of an electronic secretary 
system according to the present invention will be de- 
scribed referring to Figure 22. Incidentally., by putting like 
symbols to constituents similar to those of Figures 1 and 
9 , description is omitted. 

The present embodiment is featured by comprising 
a mode storing section 114 as mode storing means for 
storing the work environment of a user. Values set de- 
pending on the work environment of equipment are 
stored in the mode storing section 1 1 4. 



For example, in an equipment that cannot output 
voice data, a mode to output no voice is set and no voice 
is outputted when the electronic secretary acts. In this 
case, it can also be considered to output the voice con- 

s tent in letter Also, in an equipment that cannot output 
image data, a mode to output no image is set and no 
action of the electronic secretary is outputted. In this 
case, it can also be considered to output the action con- 
tent of the electronic secretary in letter. Also, it can be 

'0 considered that a different value is set for each equip- 
ment, for example, the background to be displayed for 
an electronic secretary is varied with individual equip- 
ment. 

Next, a fourth embodiment of an electronic secretary 
'5 system according to the present invention will be de- 
scribed referring to Figure 23. Incidentally, by putting like 
symbols to constituents similar to those of Figures 1, 9 
and 22. description is omitted. 

The present embodiment is featured by comprising 

20 an internal state generation section 115 for generating 
the internal state in according with a predetermined prin- 
ciple. The internal state is generated by the internal state 
generation section 115 in according with a predeter- 
mined principle. For example, at each time determined 

25 with a random number, "wink" is written into the internal 
state memory section 104. When "wink" is described in 
the internal state memory section 104, the electronic 
secretary performs an action of wink in accordance with 
Figure 2. An action of wink can be implemented by alter- 

30 nately displaying images of opened eye and closed eye. 
The timing of writing is considered to be performed at 
each predetermined time or at intervals of time deter- 
mined with a random number 

Also by comprising timing generation means, it is 

35 possible to switch the background of an electronic sec- 
retary between that of morning, that of noon and that of 
night in dependence on the time of day, or to match the 
background with spring, summer, autumn or winter in de- 
pendence on changes of the season, or to change the 

-^0 costume of an electronic secretary. 

Next, a fifth embodiment of an electronic secretary 
system according to the present invention will be de- 
scribed referring to Figure 24. 

. As shown in Figure 24, an electronic secretary sys- 

-^5 tem according to the present invention comprises: an in- 
put section 101 . such as mouse and keyboard, for input- 
ting operational information; a document data storing 
section 1 23 as storing means for storing document data 
as data to be retrieved; a retrieval section 1 22 as retrieval 

50 means for retrieving data pertinent to the retrieval con- 
dition inputted using the input section 101 from the doc- 
ument data storing section 123 distributed in a plurality 
of equipment connected through a network; an internal 
state memory section 104 for memorizing the internal 

55 state of processing: an action description storing section 
1 06 for storing the action description of an electronic sec- 
retary as an object imitating a human action: an electron- 
ic secretary action data storing section 107 for storing 



15 



EP 0 697 671 A2 



16 



data, such as image of an action and voice, ol the elec- 
tronic secretary: an electronic secretary generation sec- 
tion 105 for deciding and generating an action of elec- 
tronic secretary corresponding to the internal state out 
of the action description: a display section 108 for dis- 5 
playing an output of the retrieval section 1 22 and an out- 
put of the electronic secretary generation section 105: a 
communication interface 109 for performing data trans- 
fer with other equipment connected through a network: 
an interpreter section 110 for interpreting data to be io 
transferred and pertonming data transfer between the re- 
trieval section 122 and communication interface 109: a 
second action description storing section 1 1 2 for storing 
data describing the action of the electronic secretary in 
one or more communication partners: a second electron- is 
ic secretary action data storing section 113 for storing 
data, such as image of an action and voice, of the elec- 
tronic secretary: and a second electronic secretary gen- 
eration section 111 for generating an action of the elec- 
tronic secretary. 20 

Next, actions in the case of retrieval will be described 
along the flowchart of Figure 4. 

When a user selects a "Retrieval" button by using a 
mouse or the like (step S1 1). "Retrieval Start" is written 
into the internal state memory section 1 04 by the retrieval 25 
section 122 (step SI 2) and at the same time a window 
for keyword input is opened. When "Retrieval Start" is 
written in the internal state memory section 104. an ac- 
tion of the electronic secretary corresponding to "Re- 
trieval Start" is read out from data of Figure 2 stored in 30 
the action description storing section 106, and voice data 
"Input a keyword", stored in the electronic secretary ac- 
tion data storing section 107 is regenerated. At this time, 
the mouth of the electronic secretary moves correspond- 
ing to an output of voice data, where the movement of 35 
the mouth is implemented by an alternate display of im- 
ages of closed mouse and opened mouth during voice 
output. From this on. similarly when a voice is outputted, 
the mouth of the electronic secretary moves during voice 
output. Incidentally, the number of internal states may be 
plural. For example, it can also be set that a state of re- 
trieval start sets in while keeping a state of request re- 
ceipt. This means that the system stands in an allowable 
state of receiving a request from a user and the electronic 
secretary performs an action of retrieval start. -fs 

When a user inputs a keyword by using a keyboard 
(step SI 3), the retrieval section 122 decides whether the 
"Break" button was pressed or not (step SI 4). If the 
"Break" button is pressed, the retrieval processing is bro- 
ken off. If deciding that the "Break" button is not pressed, so 
it is decided whether the "Retrieval" button is pressed 
(step SI 5). If the "Retrieval" button is not pressed, the 
system returns to the step 1 4. Or if the "Retrieval" button 
is pressed. "Partner Selection" is written in the internal 
state storing section 104 (step S16). When "Partner Se- ss 
lection" is written. "Whom do you ask?" is regenerated 
as an action of the electronic secretary from data of Fig- 
ure 2. A "Name" button of the partner to be asked for 



retrieval is selected using a mouse or the like by a user 
(step SI 7). A mark is placed on the people selected and 
it is decided whether the "Break" button is pressed or not 
(step S18). If the "Break" button is pressed, the retrieval 
processing is broken off. Or. if the "Break" button is not ' " 
pressed, it is decided whether the "OK" button is pressed 
(step Si 9). If the "OK" button is not pressed, the system 
returns to the step S18. Or. if the "OK" button is pressed, 
a communication with other equipment starts (step S20). 

Then, when the transmit processing of requests for 
all selected communication partners ends (step S21), 
the processing of retrieval starts (step 822). Retrieval 
can be executed, for example, by a method for extracting 
a sentence in which any of keywords of sentence data 
stored in the document data storing section 123 and a 
keyword inputted by a user is coincident or sirnilar Dur- 
ing retrieval. "During Retrieval" is written into the internal 
state storing section 1 04. When the internal state is "Dur- 
ing Retrieval", an action of leafing through a book is per- 
formed as an electronic secretary's action from data of 
Figure 2. This processing is performed for all data till the 
retrieval is completed (step S23). 

Next, actions during communication will be de- 
scribed. 

As shown in Figure 5. when advancing a request 
from Equipment A to Equipment B, the internal state of 
Equipment A becomes "Communication Start Transmit" 
and a request of communication start is transmitted to 
Equipment B. At this lime, not only a request of commu- 
nication start but also the user's name and internal stale 
of Equipment A on the transmitter side are transmitted. 
In the interpreter section 110 of Equipment B on the re- 
ceiver side, a request of communication start is outputted 
to the retrieval section 1 22 and the user's name and in- 
ternal state, information related to the electronic secre- 
tary of Equipment A is outputted to the second electronic ^ 
secretary generation section 1 1 1 . In the second electron- 
ic secretary generation section 111, the action descrip- 
tion of an electronic secretary corresponding to the in- 
ternal state and data on an action of the electronic sec- 
retary responsive to the user's name are read from the 
second action description storing section 112 and sec- 
ond electronic secretary action data storing section 113 
respectively, and actions of the electronic secretary are 
generated. 

Next, a sixth embodiment of an electronic secretary 
according to the present invention will be described re- . 
ferring to Figure 25. Incidentally, by putting like symbols 
to constituents similar to those of Figure 1, description is 
omitted. 

The present embodiment is featured by comprising: 
a retrieval section 1 24 for performing a retrieval of sched- 
ule data through a network: a registry/deletion section 

1 25 for performing a registry or deletion of schedule data 
through a network: and a schedule data storing section 

126 for storing schedule data. 

Next, actions of the transmit side and receive side 
will be described along the flowcharts of Figure 26. First, 
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actions of the transmit side will be described along the 
left flowchart of Figure 26. 

When the flow of Figure 26 is called from the main 
flow of Figure 26 and a schedule setting mode sets in. 
the internal slate is turned into schedule setting by the s 
processing section 102 in the internal stale memory sec- 
lion 104 (step S51). A screen for setting a schedule as 
shown in Figure 27 is displayed on the display section 
1 08 and the system waits an input of schedule data from 
a user (step S52). When the internal slate changes, an 
electronic secretary action is generated in accordance 
with Figures 2 and 3. Incidentally, Figure 2 describes part 
of actions and does not describe some of those appear- 
ing in the flow of Figure 26. As shown in Figure 28, the 
internal state is made a communication partner of by the '5 
processing section 102 (step S53). An input of a transmit 
partner in a request of schedule is waiting for (step S54). 
As shown in Figure 29. the internal state is turned into 
communication start transmit (step S55). Request for 
communication to the equipment of a specified partner 20 
is generated (step S56). It is decided whether a connec- 
tion to the communication partner is established or not 
(step S57). If no connection with communication partner 
is established, a decision of communication connection 
is performed (step S58). As a result, if no connection was 25 
possible, the internal state is turned into absence, to a 
user is notice given through an electronic secretary that 
no connection with the partner was possible. Then, the 
internal state is turned into discontinue processing and 
notice is given to a user through an electronic secretary 30 
that to make a connection will be given up (step S59). 
Then, end processing is performed (step 360). Inciden- 
tally, when a connection was impossible, various other 
processing, such as asking a user whether request for 
reconnection is performed or not, are considered. 35 

In the above step S57, when a connection with the* 
communication partner is established, the internal state 
is turned into request transmit (step S61), and notice is 
given to a user through an electronic secretary that a re- 
quest for a current schedule is sent to the communication 40 
partner. Actually, a request for schedule is sent to the 
communication partner (step S62). The result of whether 
a request is accepted or not is waited (step S63). The 
internal state is turned into receive result (step S64), no- 
tice is given to a user through the electronic secretary ^5 
that the result of schedule request arrived from the com- 
munication partner It is decided whether the result of 
schedule request is OK or not OK (step S65). If the result 
of schedule request is OK, the internal state is turned 
into result OK and the electronic secretary is made to so 
indicate that the request was accepted (step S66). As 
shown in Figure 30, the detailed result of schedule re- 
quest is displayed (step S67). Or. if the result of schedule 
request is not OK, the electronic secretary is made to 
indicate that the request was rejected. Here, only the out- ss 
line of the currently processed content is displayed using 
voice/letter or image to a user by the electronic secretary 
and the detailed information is presented directly to a us- 



er. However, by writing not only stale information, such 
as waiting for simple request receive about internal 
slate, but also delailed information, such as date, time, 
partner and content of schedule request presented, into 
the internal state, it is also possible that an electronic 
secretary indicates detailed information collectively to a 
user. Furthermore, the end processing, such as end of 
communication accompanying the end of schedule sel- 
ling is performed (step S60). 

Next, actions on the receive side will be described 
along the right flowchart of Figure 26. 

The internal state is turned into request receive wait, 
and an electronic secretary is made to display the re- 
quest receive wait (step S71 ). The system comes to wait 
for receiving the communication request through a net- 
work (step S72). A connection request is received and a 
network is established (step S73). The internal state is 
turned into communication start receive, and the elec- 
tronic secretary is made the display of replying (step 
S74). A request for schedule is received (step S75). The 
internal state is turned into request receive and the elec- 
tronic secretary is made the display of receiving a re- 
quest (step S76). As shown in Figure 31, the details of 
the request is displayed (step S77). A user's input telling 
the intention to accept or to refuse the schedule request 
is waited for and the input is checked (step S78). If the 
request is refused, not OK is transmitted (step S79). Or. 
if accepted. OK is transmitted (step S80). The internal 
state is turned into result transmit and the electronic sec- 
retary is made the display of transmitting the results (step 
S81). The end processing, such as end of communica- 
tion, is performed (step S82). 

Next, a seventh embodiment of an electronic secre- 
tary system according to the present invention will be de- 
scribed referring to Figure 32. Incidentally, by putting like 
symbols to constituents similar to those of Figures 1 . de- 
scription is omitted. 

The present embodiment is featured by comprising: 
a history memory section 1 27 as history memory means 
for memorizing the history of the internal state to be 
memorized in the internal state rnemory section 104. In- 
cidentally, if there is a problem in that the memory ca- 
pacity becomes insufficient when the whole history is 
memorized, a ring buffer in which a predetermined 
number of histories f rorri the most recent are memorized 
is desirable. Also, inside the history memory section, 
there are a time generation section arid a consecutive 
number generation section and it is desired to memonze 
the generating time and the serial number of the internal 
slate as well as the internal state. And, on the display 
section 108, the history including detailed information 
among the internal state is displayed, for example as 
shown in Figure 33. 

Next, an eighth embodiment of an electronic secre- 
tary system according to the present invention will be de- 
scribed referring to Figure 34. Incidentally, by putting like 
symbols to constituents similar to those of Figure 1, de- 
scription is omitted. 
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The electronic secretary system comprises: an elec- 
tronic mail processing section 123 as electronic inail 
processing means for processing data interpreted as an 
electronic mail by the interpreter section 110: an elec- 
tronic mail storing section 1 29 for storing data interpreted 
as an electronic mail: a header storing section 1 30 as 
header storing means for storing the header information 
of an electronic mail letter; and a classification section 
1 31 as classification means for classifying the electronic 
mail letters stored in the electronic mail storing section 
129. 

Actions of the electronic secretary corresponding to 
the internal state stored in action description storing sec- 
tion 1 06 include not only those shown in Figure 2 but also 
those shown in Figure 35. The internal state written in 
the in ternal state memory section 1 04 is read out by elec- 
tronic secretary generation section 105. The action de- 
scription corresponding to the internal state is read from 
data stored in the action description storing section 106. 
Based on the read action description, a voice, image or 
the like stored in the electronic secretary action data stor- 
ing section 1 07 are displayed on the display section. The 
number of internal states may be plural. These actions 
are repeated at each predetermined time, or performed 
whenever the internal stales change. 

Next, the classification processing will be described 
along the flowcharts of Figures 36a and 36b. 

First, when a user presses a "mail" button (step 
S91 ). "Mail Start' is written into the internal state memory 
section 103 by the processing section 102 (step S92). 
From data stored in the action description storing section 
106 as shown in Figure 35,. the action description corre- 
sponding to the internal state is read out and a mail 
screen is displayed on the display section 108. In an 
electronic mail letter, a mail flag is set by the electronic 
mail processing section 128. The mail flag is set to 1 for 
an electronic mail letter not yet read and to 0 for an elec- 
tronic mail already read. By the electronic mail process- 
ing section 1 28. electronic mail letters having ihe mail 
flag of 1 are retrieved and the header information is dis- 
played in a list (step S93). Then, when a user presses a 
"classifying" button (step S94), "Classifying Start" is writ- 
ten into the internal state memory section 1 04 (step S95). 
From data shown in Figure 35, voice data "Classify", 
stored in the electronic secretary action data storing sec- 
tion 107, is regenerated and classifying starts. While 
classifying starts, "During Classifying" is written into the 
internal state memory section 104 (step 396) and in ac- 
cordance with data shown in Figure 35, the manner of 
an electronic secretary classifying the "header" buttons 
according to categories is displayed (step S97). 

When classifying ends, "Classifying End" is written 
into the internal state memory section 104 (step S98), 
an action of "During Classifying" is completed in accord- 
ance with data shown in Figure 35. The classified results 
are delivered to the processing section 102, classified 
according to categories and outputted from the display 
section 108 (step 899). At this time, an "End" button is 



displayed. When a user presses the "End" button (step 
SI 00). "lytail End" is written into the internal state mem- 
ory section 104 (step S101) and the mail screen closes 
in accordance with data shown in Figure 35. Or, when a 
5 user does not press the 'End' button, press of the "head- 
er" button by a user (step Si 02) causes the electronic 
mail letter corresponding to the header pressed to be re- 
trieved in the electronic mail processing section 128 and 
the content of the electronic mail letter is displayed (step 
SI 03). When the content of the electronic mail letter is 
called in the electronic mail processing section 128, the 
mail flag for the electronic mail letter is set to 0 (step 
SI 04). Then, it is decided whether a user pressed a 
"Close" button or not (step 8105). If a user presses the 
'5 "Close" button, the indication of the content distinguishes 
and the system returns to the above step SI 00. Or, if it 
was decided that a user did not press the "Close" button, 
it is decided whether a user pressed a "Reproduce" but- 
ton (step SI 06). If it was decided that a user did not press 

20 the "Reproduce" button, the system returns to the above 
step SI 05. If it was decided that a user pressed the "Re- 
produce" button, "Mail Reproduce" is written into the in- 
ternal state memory section 104 (step SI 06), and image 
data, voice data or the like contained in data of the elec- 
ts ironic mail letter are reproduced in accordance with data 
shown in Figure 35. Then, the processing to be per- 
formed when a new electronic mail arrived will be de- 
scribed along the flowchart of Figure 37. 

When the electronic mail arrives at the communica- 

30 tion interface 1 09 (step S1 1 1 ), data of the electronic mail 
is sent to the electronic mail processing section 128 
through the interpreter section 1 10. At this time, "Mail Ar- 
rival" is written into the internal state memory section 104 
(step S112) and in accordance with data stored in the 

•35 action description storing section 1 06 as shown in Figure 
35, an action of the electronic secretary receiving a post- 
al card is outputted from the display section 1 08. An elec- 
tronic mail is stored in the electronic mail storing section 
129 by the electronic mail processing section 128 (step 

"to S 1 1 3) and the header information is stored in the header 
storing section 130 (step S1 14). The mail flag of this elec- 
tronic mail is set to 1 (step S1 15). "N/lail Unprocessed" is 
written into the internal state storing section 104 (step 
8116) and in accordance with data stored in the action 

^5 description storing section 106 as shown in Figure 35, a 
postal card is displayed on the desk of an electronic sec- 
retary. So long as an electronic mail having the mail flag 
of 1 is present in the electronic mail processing section 
128, the internal slate remains "f^ail Unprocessed" and 

50 the display of the postal card is kept to be outputted. 

Next, actions of classifying will be described along 
the flowchart of Figure 38. 

When a user presses a "Classify** button while the 
header information of electronic mail having the mail flag 

55 of 1 is displayed in the list, first, data of the first electronic 
mail letter are retrieved from the electronic mail storing 
section 1 29 and the component vector is calculated (step 
Si 21). Incidentally, categories of classifying are set by a 
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user in advance, a reference vector is calculated tor each 
category, and the data shall be possessed by the clas- 
sifying section 131 . A similarity between the component 
vector of an electronic mail letter and the reference vec- 
tor of each category is calculated (step Si 22). In evalu- s 
aling the similarity, various methods, such as by comput- 
ing the inner product or difference of vectors, are con- 
sidered, but either method is possible. If the similarity is 
calculated concerning all categories (step Si 23). the 
name of the highest category in similarity and a value of 'O 
similarity are extracted (step 8124). 

The display of "header' buttons is so modified as to 
easily find the relevant category from the extracted cat- 
egory names in the display section 108 (step Si 25). For 
example, such processing as collectively displaying '5 
those of the same category or distinguishing different 
categories by color, is performed. Also, the similarity is 
simultaneously displayed (step SI 26). or the degree of 
closeness to a certain category is made clear. These cal- 
culations are performed for all listed electronic mail let- 20 
ters. When all calculations end (step SI 27), the classi- 
fying end flag is turned into 1 (step 8128). 'Classifying 
End' is written into the internal state storing section 104 
and in accordance with data stored in the action descrip- 
tion storing section 106 as shown in Figure 35. the etec- 25 
tronic secretary finishes the action of "During Classify- 
ing". Incidentally, not only an electronic mail letter not yet 
read but also those already read can be also classified. 
The processing of this case is also performed like the 
above. 

Next, the classifying processing will be described 
using a concrete example. 

As shown in Figure 39, an electronic mail letter com- 
prises an electronic mail text and a "header" button. Fig- 
ure 39 (b) shows a displayed example of 3 items, a trans- 35 
mitter, a received date and the content, extracted as ' 
header information. As shown in Figure 40, the catego- 
ries for classifying shall be set by a user. In the classifying 
section 131, the component vector is calculated con- 
cerning the electronic mail letter of Figure 39 (a). The -^o 
similarity between this component vector and the refer- 
ence vectors of three categories in Figure 40 is respec- 
tively evaluated. As shown in Figure 41 (a), it is obtained 
as the calculated results that the similarity is 66 for the 
category 1 (VR relation). 27 for the category 2 (neuro -fs 
relation) and 50 for the category 3 (others). The highest 
sim ilarity among them is a similarity of 66 for the category 
1 , and accordingly this letter is classified into the cate- 
gory 1 . so that the category 1 and similarity 66 are ex- 
tracted. Figure 41b shows the extracted results. Seeing 50 
the result as indicated in Figure 41 (b), the display sec- 
tion 108 displays "header" buttons shown in Figure 39 
(b) at the category 1 as shown in Figure 41c, where 
"header" buttons of the electronic mail classified accord- 
ing to categories are displayed in the respective catego- 55 
ries. A "score" indicated by the side of a "header" button 
represents the similarity. In Figure 41c. the arrangement 
of electronic mail letters in each category is in the order 



of arrival. As shown in Figure 4 Id, a method for display 
electronic mail letters in sequence from the highest sim- 
ilarity can be also considered. In this case, it is advisable 
that after category names and similarity are extracted for 
air electronic mail letters, values of similarity are com- 
pared for the electronic mail letters belonging to the 
same category and the letters are arranged in sequence 
from the highest similarity in each category. 

Next, the above example of action will be described 
in an actual display screen. 

When a user presses a "fV/lair button, a mail screen 
as shown in Figure 42 is opened, the list of header infor- 
mation in electronic mail letters not yet read is displayed. 
Hereupon, when pressing a "Classify" button, the man- 
ner of an electronic secretary to classify the "header' but- 
tons is displayed as shown in Figure 43. Figure 44 shows 
the classified results, where electronic mail letters are 
separately displayed under classification by categories 
of their "header" buttons. The item indicated by scores 
is a similarity, while classified categories and similarity 
are indicated. Herein, when a user presses a definite 
"header" button, a window for showing the content of 
electronic mail tetters corresponding to a pressed "head- 
er" button is displayed as shown in Figure 45. At this 
time, a check mark is put on the pressed "header" button, 
which indicates the relevant electronic mail letter has 
been read. When pressing a "Reproduce" button denot- 
ed by a picture of speaker, image data, audio data or the 
like are reproduced. When pressing a "Close" button, the 
mail screen closes and the processing of electronic mail 
letters ends. As show/n in Figure 46, an example of dis- 
play when a new electronic mail arrived displays a postal 
card receiving manner of an electronic secretary. As 
shown in Figure 47. an example of display when an elec- 
tronic mail having the mail flag of 1 is included in the elec- 
tronic mail storing section 129 displays an image that a 
postal card is placed on the desk of the electronic sec- . 
retary, which informs a user of electronic mail arrival. 

Next, a ninth embodiment of an electronic secretary 
system according to the present invention will be de- 
scribed referring to Figure 48. Incidentally, by putting like 
symbols to constituents similar to those of Figure 34, de- 
scription is omitted. 

The present embodiment is featured by comprising 
the classify mode storing section 1 32 as classify mode 
storing means for storing the classify mode in accord- 
ance with an instruction to be inputted from the input sec- 
tion 101. In the classify mode storing section 1 32. values 
set on classify modes in the classify section 131, esti- 
mating function, and method for indicating the classified 
results are stored. 

Next, the procedure for setting a classify mode will 
be described along the flowchart of Figure 49. 

When a user selects the 'Classify f^ode Setting" 
(step SI 31), "Classify Mode Setting" is written into the 
internal state memory section 104 (step 8132). In ac- 
cordance with data stored in the action description stor- 
ing section 106 as shown in Figure 35, a screen for set- 
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ting a classify mode is displayed and the list of classify 
modes stored in the classify mode storing section 1 32 is 
displayed. As shown in Figure 50, examples of classify 
modes include "By Conlenf. classifying by the content 
of electronic mail letters: "By Transmitter", classifying by s 
transmitters: "By Group", classifying by the belonging 
group of transmitters: and "By Date", classifying by the 
received date of electronic mail letters. It is advisable 
that, by preparing the item of "User Setting", a user him- 
self can set a classify mode. When a classify mode is io 
selected by a user from the list of classify modes (step 
S1 33), an alternation in classify mode is performed in the 
classify mode memory section 132 (step SI 34). There- 
after, an alteration in estimation function is performed ac- 
cording to a set classify mode (step S 1 35). is 

Figure 51 shows an example of estimation function. 
In the mode 1. "By Content", as described in a first em- 
bodiment, the similarity between the component vector 
of a sentence and the reference vectors is calculated and 
the relevant mail letter is classified in a category of the 20 
highest similarity. In the mode 2, "By Transmitter", one 
letter is picked up from among the electronic mail to be 
classified and the similarity with others is calculated. In 
this case, seeing the header information, to the similarity 
is assigned 1 for electronic mail letter of the same trans- 25 
mitter and 0 for that of a different transmitter, where the 
electronic mail having a similarity of 1 are collectively 
classified in one and the same category. This operation 
is continued till the whole electronic mail is classified. In 
the mode 3. "By Group", after registering the members 30 
of each group in advance, electronic mail letters are clas- 
sified referring to a name of transmitter among the head- 
er information into belonging groups. In the mode 4, "By 
Date", seeing dates of the header information as with the 
mode 2, to the similarity is assigned 1 for those of the 35 
same date and 0 for others, where the electronic mail 
having a similarity of 1 are collectively classified in one 
and the same category. 

In relation to the memorized classify mode, an esti- 
mation function is set in advance but can be altered by -to 
a user. Then, returning to Figure 49, a display method is 
altered in accordance with a classify mode (step 3136). 
In dependence on a classify mode, examples of display 
methods are as follows: In the mode 1 . "By Content", 
"header"* buttons are collected in display for each cate- -is 
gory as shown in Figures 41c and 41 d. In the mode 2. 
"By Transmitter^ "header" buttons are listed in display 
for each transmitter as shown in Figure 52a. In the mode 
3. "By Group^ "header* buttons are divided in display for 
each group as shown in Figure 52b. In the mode 4, "By so 
Date", "header" buttons are divided in display for each 
date as shown in Figure 52c. Similarly in other modes, 
a display method is set for each mode. Thereafter, the 
display of similarity is set. If set to "the similarity is not 
displayed", such can be set that the similarity is not dis- 55 
played. Meanwhile, when the similarity represents the 
closeness to the relevant category as withtheeighth em- 
bodiment, an electronic mail letter having a higher simi- 



larity may be regarded as high in priority. At this time, it 
can be set in such a manner as to display the electronic 
mail in sequence from the highest similarity. 

Furthermore, in the above step Si 33 of Figure 49, 
when selecting a 'User Setting", a user sets a classifying 
method from the input section 101 (step Si 37). Hereup- 
on, setting of an estimation function (step Si 38) and set- 
ting of a display method (step S 1 39) are respectively per- 
formed. The classify mode set by a user is registered in 
the classify mode storing section 132 (step Si 40), to 
which a mode number is assigned. In this way, if a clas- 
sify mode is set, "f^ode Setting Completion" is written 
into the internal stale storing section 104 (step SI 41). 
The screen for setting a classify mode closes and the 
processing ends. In selection of a classify mode, when 
selecting "Discontinue", an alteration in classify mode is 
not performed and the processing ends. 

Actions to be performed when a user presses a 
"Classify" button were described, but such setting is 
thinkable that classification can proceed when a user 
presses a "Mail" button. In this case, the classified results 
are arranged to be displayed instead of the list of elec- 
tronic mail. 

This invention extends to the disclosed methods of 
facilitating communication at the man-machine interface 
of an office data processing system or electronic secre- 
tary system. 

The invention may be embodied in other specific 
forms without departing from its scope as defined by the 
claims. 



Claims 

1. An electronic secretary system for performing a 
series of processing through an interaction between 
a user and an electronic secretary displayed as an 
object imitating a human action, characterized by 
comprising: 

input means (101) for inputting information, 
such as user's instruction; 

processing means (102) responsive to the 
input information for perfonning a predetermined 
processing; 

data storing means (1 03) for storing data to be 
processed: 

internal state memory means (104) for mem- 
orizing the internal state of processing: 

first action description storing means (106) for 
storing the action of said electronic secretary 
described corresponding to the state of said 
processing; 

first electronic secretary generation means 
(105) for generating actions of said electronic sec- 
retary from the state of processing memorized in 
said internal state memory means and the descrip- 
tion stored in response to the state of processing into 
said action description storing means; and 
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display means (108) for performing an action 
of said electronic secretary and a display of data. 

2. The electronic secretary system as set forth in Claim 

1, characterized by further comprising; 5 

means (109) for communicating with other one 
or more electronic secretary systems; 

second action descriptbn storing means (112) 
for storing data describing the action of the elec- 
tronic secretary in a communication partner; io 

second electronic secretary generation means 
(1 11 ) for generating an action of said electronic sec- 
retary in the communication partner 

3. The electronic secretary system as set forth in Claim '5 
1 or Claim 2. characterized by further comprising; 

mode memory means (114) for memorizing 
the user's work environment. 

4. The electronic secretary system as set forth in any 20 
of Claim 1 to Claim 3, characterized by further com- 
prising; 

internal state generation means (115) for gen- 
erating the internal state in accordance with a pre- 
determined rule. 25 

5. The electronic secretary system as set forth in any 
of Claim 1 to Claim 4. characterized in that 

said processing means (102) retrieves data 
corresponding to the retrieval condition inputted by 30 
a user from the electronic secretary system of said 
user himself or other electronic secretary system 
connected through a communication means. 



processing an electronic mail sent/received through 
said communicator means; and 

classifying means (131 ) for classifying an elec- 
tronic mail processed by said electric mail means. 

9. The electronic secretary system as set forth in Claim 
8. characterized by further comprising; 

classifying mode memory means (132) for 
memorizing a classification mode by referring to the 
instruction inputted from said input means. 

10. An office data processing system comprising an 
input section for inputting information, a processing 
section for performing predetermined processing 
operations in accordance with the input information, 
and a display section for displaying an output of the 
processing section, characterised by an action 
description data storing section for storing action 
description data relating to respective processing 
operations of the processing section, and an action 
description generation section, responsive to the 
particular processing operation performed by the 
processing section, for generating and displaying on 
the display section, using the related action descrip- 
tion data, a pictorial display of the office environment 
and/or a human figure identifying the processing 
operation. 

11. An office data processing system, characterised by 
means for generating and displaying a pictorial dis- 
play of the office environment and/or a human figure 
identifying the type of current or past processing 
operation performed by the system. 



6. The electronic secretary system as set forth in any 35 
of Claim 1 to Claim 4. characterized in that 

said processing means (102) performs the 
registration, deletion or retrieval of a schedule in the 
electronic secretary system of said user himself or 
other electronic secretary system connected -^o 
through a communication means. 

7. The electronic secretary system as set forth in any 
of Claim 1 to Claim 4. characterized by further com- 
prising; ^5 

history memory means (127) for memorizing 
the history of the internal state; wherein 

in response to request from a user, what action 
the electronic secretary has made is displayed on 
said display means by referring to the history mem- so 
orized on said history memory means. 

8. The electronic secretary system as set forth in Claim 
1 . characterized by further comprising; 

communicator means (109) for communicat- 55 
ing with other one or more electronic secretary sys- 
tems; 

electronic mail processing means (128) for 



1 2. An office data processing system according to claim 
1 1 . capable of communication with a second office 
data processing system, the first system comprising 
means for generating and displaying a pictorial dis- 
play of the office environment and/or a human figure 
identifying the type of processing operation per- 
formed by the second system during said communi- 
cation. 
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FIG. 2 
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FIG. 7 
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FIG. 35 
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(a) 



Report on official tour abroad: Aug. 18, 1993: T. Hashimoto 

Object: Through an event of CG, visit to SIGGRAPH and visit to 
Silicon Graphics, maker of graphics workstations, to 
perform a research for the trends of virtual reality. 

Interval: Aug. 1-8, 1993 

Outline: 

The greatest topic for this time in SIGGRAPH was the newest 
multi-workstation of Silicon Graphics. It is a graphics 
workstation, nonnally loaded with a TV conference function, 
priced as low as a million yen, and having Unix employed as OS. 

The second topic was a virtual reality. A demonstration of CG 
capable of being rounded for inspection was being held by 
Silicon Graphics as well as Kaiser, Evans and Satherland, 
Division, Sense 8, Polhemus, MuliGen, and other many makers. 

As a display, boom mount type displays were mostly used, but 
Silicon Graphics, Kaiser, Division, and Polhemus used their 
original head-mount type displays. 

My general impression of virtual reality is firstly that the 
image quality is improved by leaps and bounds. The appearance 
of a new graphics engine, Reality Engine, of Silicon Graphics 
IS said to greatly contribute to this. Secondly, in contrast to 
a good image quality, a depicted world contradicts the physical 
laws of a real world in plenty of points, and not a few elements 
seemed deficient in a spirit of services as entertainments. 

I felt many problems yet remain to be solved on a method for 
interacting with a virtual world, called interactivity, and a 
method for describing the physical laws of a virtual world. 
Would you please see the photographed scenes of event sites. 



(b) 



Transmrtter Date Contftnt 
□ Hashmoto 9/22 Official SKSGRAPH tour | 



54 



EP 0 697 671 A2 



FIG. 40 





CATEGORY 


1 


VR RELATED 


2 


NEURO RELATED 


3 


OTHERS 



FIG. 41A 



CATEGORY 


11 


2 


3 


SIMILARITY 




27 


50 



FIG. 41 B 



1 : 66 



55 



EP 0 697 671 A2 



FIG. 41C 



SCORE 











□ Hashnnoto 9/22 


Official SIGGRAPH tour| | 66 | 


□ Hiide 9/22 


Research inst visit 




\\^\ 


□ Aranaki 9/23 


Exhibition report 








Neuro related ^— ^ 


□ Matsuoka 9/21 


Chaos engineering 




|a2| 


□ Sakamoto 9/21 


SYNAPS 




|56| 


□ Inoue 9/23 


Neuro lecture in turn 




|85| 



Others 



□ Toirita 9/21 TDL 



69 



56 



EP0 697 671 A2 



FIG. 41D 





VR related 






SCORE 




h 








□ Aramaki 


9/23 


Exhibition report 


\\^\ 


□ Hiide 


9/22 


Research inst visit 


\\^\ 


□ Hashmoto 9/22 


Official SIG6RAPH tour 


II »l 














Meuro related ] 








□ Inoue 


9/23 


Neuro lecture in turn 


|85| 


□ Matsuoka 9/21 


Chaos engineering 


|62| 




□ Sakanoto-.9/21 


SYNAPS 1 


l»l 














Others 












□ Tofrita 


9/21 


TDL 1 


69 





57 



EP 0 697 671 A2 




58 



EP 0 697 671 A2 



o 
<_> 

CO 



iS 



CO 
LL. 



CO 
TO 

o 



lOJ 



in 



<3 



-a 
15 



c 

<D 



15 

ru 



CO 



CO 

ru 



























a> 






t= 
o 












IS 


<I> 




X 
UJ 






S3 


S3 






























cr 




n 


n 


n 




"£ 

O 
I— 

loJ 



59 



EP 0 697 671 A2 




60 



EP 0 697 671 A2 




61 




62 




63 



EP 0 697 671 A2 




CD 



ljjZO 



eg 

CO 



==LU20 



£55 




64 



EP 0 697 671 A2 



FIG. 49 



Q START ) 



A USER SELECTS 
■ CUSSIFY MODE 
SETmC. 



— S131 



TURN THE fflTERNAL 
STATE INTO " CLASSIFY 
MODE SETTING'. 



■ DISCONTINUE- 



c 



END 



J 




SI 32 



USER SEHING- 



S137 



ALTER THE 
CLASSIFY MODE 



S135 



S136 



SE1 
CLASS 
MET 


r A 

IFYING 
HOD. 


SI 37 





ALTER THE 
ESTIMATION 
FUNCTION 



SET AN 
ESTIMATION 
FUNCTION 



ALTER THE 
DISPLAY METHOD. 



S141 



SI 37 

.S 



SET A DISPLAY 
METHOD. 



SI 36 



REGISTER A 
CLASSIFY MODE. 



TURN THE INTERNAL 
STATE INTO " MODE 
SETTING END". 



c 



END 



3 



65 



EP 0 697 671 A2 



FIG. 50 





CLASSIFY MODE 


1 


BY co^^^ENT 


2 


BY TRANSMITTER 


3 


BY GROUP 


4 


BY DATE 






OTHERS 


USER SETTING 



FIG. 51 



MODE 


EVALUATION FUNCTION 


1 




2 




3 


TO thom'ot' others'''* ™^ °' ™ " 


4 


Wa^OFOmERS™ ' ™ ™^ ""^ ™^ AND TO 0 FOR 







66 



EP 0 697 671 A2 



C 
C7> 

cz 

CD 
CO 

\ 



CO 

inJ 



n 




67 



EP O 697 671 A2 



^ loojia 



c 
o 



cn 
o 

<D 

a> 

cr 

■o 
c 

CM 



o 



TO 

o 

\ 

cn 



lElJ 



■G 
-d 

O 

o 

Q. 
5 



i 



3§ 



cr 



X 

cn 



8. 



TO 



\ 



-8 



68 



EP 0 697 671 A2 



O 

d 



-8 

CD 

ZD 

a 



nJia 



\ 



CO 
TO 

o 



a 



o 

a 



69 



THIS PAGE BLANK (uspto) 



(19) 



J 



Europaisches Patenlamt 
European Patent Office 
Office europeen des brevets 




(12) 



(11) EP 0 697 671 A3 

EUROPEAN PATENT APPLICATION 



(88) Dale of publicalion A3: 

30.07.1997 Bulletin 1997/31 

(43) Date of publication A2; 

21,02.1996 Bulletin 1996/08 

(21) Application number; 95305582.9 

(22) Dale of filing: 10.08.1995 



(84) Designated Contracting States; 
DE GB 

(30) Priority: 11.08.1994 JP 189519/94 

(71) Applicant: SHARP KABUSHIKI KAISHA 
Osaka 545 (JP) 

(72) Inventors: 
♦ Ueda, Toru 

Ichihara-shi, Chiba-ken (JP) 

(54) Electronic secretary system 

(57) The electronic secretary system comprises an 
input section (101) for inputting operation information* a 
processing section (102) for performing a predeter- 
mined processing in accordance with the input informa- 
tion; a data storing section (103) for storing the input 
(nformation; an internal state memory section (104) for 
memorizing the state of processing; an action descrip- 
tion storing section (106) for storing the action descrip- 
tion of an electronic secretary: an action data storing 



(51) intciA G06F 17/60 



• Matsuoka, Yasuko 

Chiba-shi, Chiba-ken (JP) 

(74) Representative: Brown, Kenneth Richard et al 
R.G.C. Jenkins & Co. 
26 Caxton Street 
London SW1H ORJ (GB) 



section (107) for storing data, such as an action image 
a voice, or the like of the electronic secretary; an elec- 
tronic secretary generation section (105) for deciding 
and generating an action of the electronic secretary in 
accordance with the internal state; a display section 
(108) for displaying an output of the processing section 
(102) and an output of the electronic secretary genera- 
tion section (105); an interface (109) for transmitting da- 
ta with other equipment; and an interpreter section 110 
for interpreting the transmitted data. 



109 7 IK 



FIG. 1 



COMMUNICATION 
INTERFACE 



10 



II^RPRETER 
SECTION 



101 



INPUT 
SECTION 



102 



108 



OISPL>VY 
SECTION 



104 



-I PROCESSING 
SECTION 



103 

_ ^ 



INTERNAL 

STATE 
MEMORY 
_ SECTION 



105 



ELECTRONIC 
SECRETARY 
GENERATION 
SECTION 



106 



DATA 
STORING 
SECTION 



ACTION 
DESCRIPTION 
STORING 
SECTION 



107 



ELECTRONIC 
SECRETARY 
ACTION DATA 
STORING SECTION 



Printed by Jo<iv-r 



/'jyji PARIS {FR I 



EP 0 697 671 A3 



Curnpean FalcnC 
Office 



EUROPEAN SEARCH REPORT 



Application rsDtnb<r 

EP 95 30 5582 



DOCUMENTS CONSIDERED TO BE RELEVANT 



C Had on of documcBC with indication , where appropriate, 
of relevant passages 



Relevant 
to clatin 



CLASSIFICATION OF THE 
APPU CATION (lnt.CL&) 



EP 0 562 995 A (IBM) 29 September 1993 



* the whole document * 

WO 91 01022 A (RAMER AND ASSOCIATES INC) 
24 January 1991 

* page 3, line 24 - 1 ine 29 * 

EP 0 438 017 A (IBM) 24 July 1991 

* the whole document * 

WO 93 14844 A (BEST ROBERT M) 5 August 
1993 

* abstract; claims 1-8; figures 1-7 * 
COMPUTER. 

vol. 24, no. 9, 1 September 1991, 
pages 49-57, XPOO026446O 
MARCUS A ET AL: "USER-INTERFACE 
DEVELOPMENTS FOR THE NINETIES" 

* page 55, column 1, line 44 - page 57, 
column 1, line 46 * 



The present search report has been drawn up for all dftuns 



1,3-5,7, 
9-12 
2,6,8 



2,6,8 



1,3-5,7, 
9-12 



1.3-5,7, 
9-12 



G06F17/60 



1-12 



TECHNICAL FlfXOS 
SKARCHU) (Iat.CX6) 



G06F 



THE HAGUE 



3 June 1997 



Suendermann, R 



CATECORY OF CTIEn DOCUMENTS 

X : pirticulirty rele^nt if taken aJone 

Y ; partioilvly relenot if combiooJ with anotbcr 

documeai of the same cifegory 
A : teetinologicil background 
O : oon-writien disdosure 
P ; inicnii«4ia{« document 



T : theory or prtodple underlying the invention 
E : earlier pifent docutnent, but published on. or 

after tbe filing date 
U : document dted in the zpplicatton 
L : document dted for other reasons 



: member of (he same patent funily, corresponding 
document 



2 



